Skip to content

Add interactive text-query annotation button#1708

Merged
BryonLewis merged 14 commits into
mainfrom
dev/text-query-annot-button
Jul 3, 2026
Merged

Add interactive text-query annotation button#1708
BryonLewis merged 14 commits into
mainfrom
dev/text-query-annot-button

Conversation

@mattdawkins

Copy link
Copy Markdown
Member

Summary

Follow-up to #1582. Adds the SAM3 interactive text-query annotation button on top of the interactive segmentation + stereo branch.

Changes

  • EditorMenu text-query button + dialog (query text, threshold, all-frames toggle).
  • textQuery / refineDetections / runTextQueryPipeline frontend API + IPC handlers.
  • Text-query methods on the consolidated interactive service.
  • Viewer / ViewerLoader wiring (text-query-init / text-query / text-query-all-frames, service-ready forwarding).

Notes

@mattdawkins mattdawkins force-pushed the dev/add-interactive-seg-and-stereo branch from 7858f2f to 20303f0 Compare June 22, 2026 16:08
@mattdawkins mattdawkins force-pushed the dev/text-query-annot-button branch 2 times, most recently from 37bc67d to fc8e213 Compare June 22, 2026 16:25
Base automatically changed from dev/add-interactive-seg-and-stereo to main June 27, 2026 16:07
SAM3 text-query button, dialog, API, and IPC handlers, on top of the
merged interactive segmentation + stereo work on main.

The button lives in the annotation type list (with the 't' shortcut)
alongside rectangle/polygon/line/segment, and opens a dialog to run a
SAM3 text query on the current frame or, optionally, across all frames
as a pipeline job. Desktop-only: gated behind a textQueryEnabled prop
set only by the desktop ViewerLoader, so the web client renders nothing.

Brings the feature to parity with origin/viame/master (SAM3 attribution
text, persistent dialog while loading, exit edit mode on completion).
@mattdawkins mattdawkins force-pushed the dev/text-query-annot-button branch from fc8e213 to 25ad49b Compare June 30, 2026 00:41
Route the single-frame text query through the correctly-named text-query-submit event so the submit handler actually runs and surfaces failures instead of silently doing nothing. Resolve the frame's media path and frame time via the segmentation resolvers so text query works on video and multicam datasets, not just image sequences.
@mattdawkins mattdawkins requested a review from BryonLewis July 1, 2026 13:48
Adds an off-by-default 'Replace existing annotations' toggle to the text
query dialog. On the current-frame path it clears the current frame's
annotations before applying the results; on the all-frames path it forwards
replace_existing to the SAM3 refiner pipeline. Also corrects the all-frames
pipeline file name and adds the refiner 'sam3:' param prefixes so the query
text and threshold actually reach the refiner.
The interactive (current-frame) text query ran with no feedback while the
model processed the frame. Add a progress dialog shown for the duration of
the single-frame query; the all-frames path is unaffected since it runs as a
pipeline job with its own progress UI.
Single-frame text query created every track with the default confidence of
1.0 because TrackStore.add() seeds the confidence pair at 1.0 and the handler
never applied the detection's score. Call setType with det.score so the
model's confidence is reflected instead of always showing 100%.
When a job process exits non-zero (and was not cancelled), scan its captured
output for lines beginning with 'ERROR:' and show them in a dialog. Processes
emit a single 'ERROR: <cause>' line to report the real failure reason (e.g. a
GPU out-of-memory during SAM3 model load), so the user sees the cause without
opening the job log.
@mattdawkins mattdawkins force-pushed the dev/text-query-annot-button branch from 021d883 to 4ddb7b1 Compare July 1, 2026 19:22
mattdawkins and others added 5 commits July 1, 2026 16:11
Text queries replace existing annotations by default instead of accumulating;
users can still uncheck it in the dialog to add alongside existing results.
Opening the text-query dialog and running a text query previously called
initialize(), which warms the point-segmentation model. Text query uses its
own (often different/heavier) model, so start the service process only
(ensureStarted) and let the text-query model load lazily on first query.
runTextQueryPipeline passed the refiner settings as a flat pipelineParams
object, but runPipeline() only threads pipelineParams.kwiverParams into the
runner's -s flags -- so text_query/replace_existing/detection_threshold were
silently dropped and the refiner ran with its defaults (querying 'object', not
the user's text). Nest them under kwiverParams.

@BryonLewis BryonLewis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was able to test this morning and I could get through the workflow properly.
In the future I may want to start dividing some other stuff up to reduce ViewerLoader and EditorMenu to make them smaller but it isn't needed for this PR.

I made a small change to add the confidence number to the label because only being seen in the thumb label when moving didn't seem right.

@BryonLewis BryonLewis merged commit 5af96b0 into main Jul 3, 2026
3 checks passed
@BryonLewis BryonLewis deleted the dev/text-query-annot-button branch July 3, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants